Remote direct memory access (RDMA) optimized high availability for in-memory data storage

ABSTRACT

A method for RDMA optimized high availability for in-memory storing of data includes receiving RDMA key-value store write requests in a network adapter of a primary computing server directed to writing data to an in-memory key-value store of the primary computing server and performing RDMA write operations of the data by the network adapter of the primary computing server responsive to the RDMA key-value store write requests. The method also includes replicating the RDMA key-value store write requests to a network adapter of a secondary computing server, by the network adapter of the primary computing server. Finally, the method includes providing address translation data for the in-memory key-value store of the primary computing server from the network adapter of the primary computing server to the network adapter of the secondary computing server.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. application Ser. No.14/709,600, filed May 12, 2015, which is a Continuation of U.S.application Ser. No. 14/525,207, filed Oct. 27, 2014, the entirety ofeach which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to the in-memory storage of data in anapplication server and more particularly to supporting high availabilitywhile storing application related data in-memory in an applicationserver.

Description of the Related Art

An application server is a software system executing upon a hardwareplatform that exposes business logic to client applications throughvarious protocols such as the hypertext transfer protocol (HTTP). Whilea Web server mainly support the transmitting of hypertext markuplanguage (HTML) defined pages to requesting browser clients for displayin a browser, an application server provides access to business logicfor use by client application programs. In this regard, the applicationprogram can use supplied business logic just as it would call a methodon an object internally disposed on a supporting client device.

In most cases, an application server exposes its business logic througha component application programming interface (API) and the applicationserver manages its own resources. Therefore, the application server alsoprovides gate-keeping services including security, transactionprocessing, resource pooling, and messaging. The application server alsomust provide performance-enhancing services such as an in-memory store.Finally, like a Web server, an application server may also supportscalability and fault-tolerance including high availability.

Advanced application servers provides for in-memory storing of data tosupport lightning fast data access. In-memory storing can be providedglobally to all logic resources of the application server, or at thecontainer level so as to support only a subset of the instances of logicresources in the application server. Recent advances in in-memory storesfor application servers utilize direct memory access (DMA) techniques.One such technique includes remote DMA. In computing, remote directmemory access (RDMA) is a direct memory access from the memory of onecomputer into that of another without involving either one's operatingsystem.

Thus, RDMA permits high-throughput, low-latency networking, which isespecially useful in massively parallel computer clusters. RDMA supportszero-copy networking by enabling the network adapter to transfer datadirectly to or from application memory, eliminating the need to copydata between application memory and the data buffers in the operatingsystem. Such transfers require no work to be done by the centralprocessing units (CPUs), key-value stores, or context switches, andtransfers continue in parallel with other system operations. When anapplication performs an RDMA Read or Write request, the application datais delivered directly to the network, reducing latency and enabling fastmessage transfer.

Distributed key/value pair store systems that exploit one-sided RDMAsuch as those found in an in-memory store engine of an applicationserver can directly read from and write to the server's memory. Thisdirect memory access is performed by utilizing RDMA between the networkadapter of the server and the memory of the server without involving theCPU or CPUs of the server. Consequently, ultra high throughput and ultralow latency results. However, high availability remains an importantproblem for one-sided RDMA access. Because the server processor is notinvolved there are no software-level hooks for high availabilityreplication, which leaves the key-value store vulnerable to hardwarefailures. Further, involving the server processor even minimally cancause dramatic performance degradation, measured in millions of requestsper second. Therefore software-based high availability schemes areundesirable when seeking performance through RDMA supported in-memorystores.

BRIEF SUMMARY OF THE INVENTION

Embodiments of the present invention address deficiencies of the art inrespect to in-memory stores in an application server and provide a noveland non-obvious method, system and computer program product for RDMAoptimized high availability for in-memory storing. In an embodiment ofthe invention, a method for RDMA optimized high availability forin-memory storing of data is provided. The method includes receivingRDMA key-value store write requests in a network adapter of a primarycomputing server directed to writing data to an in-memory key-valuestore of the primary computing server and performing RDMA writeoperations of the data by the network adapter of the primary computingserver responsive to the RDMA key-value store write requests. The methodalso includes replicating the RDMA key-value store write requests to anetwork adapter of a secondary computing server, by the network adapterof the primary computing server. Finally, the method includes providingaddress translation data for the in-memory key-value store of theprimary computing server from the network adapter of the primarycomputing server to the network adapter of the secondary computingserver.

In one aspect of the embodiment, in response to a determination that theprimary computing server has failed, an RDMA key-value store readrequest is received from a requesting client in the network adapter ofthe secondary computing server in respect to data stored in thein-memory key-value store of the primary computing server. Thereafter,because there are no guarantees that the data in the primary andsecondary computing servers are written into the same address in bothservers, an address for the data stored in the in-memory key-value storeof the primary computing server is translated to an address in anin-memory key-value store of the secondary computing server utilizingthe address translation data and an RDMA key-value store read operationis performed at the translated address. Finally, the data produced bythe RDMA key-value store read operation is returned by the networkadapter of the secondary computing server to the requesting client.Optionally, a local address table of the network adapter of thesecondary computing server is translated based upon the addresstranslation data all subsequent RDMA key-value store requests for datain the in-memory key-value store of the primary computing server areprocessed in the network adapter of the secondary computing serverutilizing the translated local address table.

In another aspect of the embodiment, an RDMA key-value store updaterequest is received in the network adapter of the primary computingserver directed to data stored in the in-memory key-value store of theprimary computing server. Thereafter, in response to the RDMA key-valuestore update request, an RDMA key-value store update operation isperformed on the stored data by the network adapter of the primarycomputing server. As before, the RDMA key-value store update request isreplicated to the network adapter of the secondary computing server, bythe network adapter of the primary computing server and the addressingfor the stored data in the in-memory key-value store of the primarycomputing server is translated to an address in an in-memory key-valuestore of the secondary computing server utilizing the addresstranslation data. Finally, the stored data is updated in the in-memorykey-value store of the secondary server utilizing the translatedaddressing.

In another embodiment of the invention, an application server dataprocessing system is configured for RDMA optimized high availability forin-memory storing. The system includes a primary computing server with acorresponding in-memory key-value store and at least one processor andnetwork adapter, and also a secondary computing server with acorresponding in-memory key-value store and at least one processor and anetwork adapter. The system also includes an RDMA optimized highavailability module disposed in each of the network adapters. The moduleincludes program code enabled to receive RDMA key-value store writerequests in the network adapter of a primary computing server directedto writing data to the in-memory key-value store of the primarycomputing server, to perform RDMA write operations of the data by thenetwork adapter of the primary computing server responsive to the RDMAkey-value store write requests, to replicate the RDMA key-value storewrite requests to the network adapter of the secondary computing server,by the network adapter of the primary computing server, and to provideaddress translation data for the in-memory key-value store of theprimary computing server from the network adapter of the primarycomputing server to the network adapter of the secondary computingserver.

Additional aspects of the invention will be set forth in part in thedescription which follows, and in part will be obvious from thedescription, or may be learned by practice of the invention. The aspectsof the invention will be realized and attained by means of the elementsand combinations particularly pointed out in the appended claims. It isto be understood that both the foregoing general description and thefollowing detailed description are exemplary and explanatory only andare not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute partof this specification, illustrate embodiments of the invention andtogether with the description, serve to explain the principles of theinvention. The embodiments illustrated herein are presently preferred,it being understood, however, that the invention is not limited to theprecise arrangements and instrumentalities shown, wherein:

FIG. 1 is a pictorial illustration of a process for RDMA optimized highavailability for in-memory storing;

FIG. 2 is a schematic illustration of a application server dataprocessing system configured for RDMA optimized high availability forin-memory storing; and,

FIGS. 3A through 3D, taken together, are a flow chart illustrating aprocess for RDMA optimized high availability for in-memory storing.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the invention provide for RDMA optimized highavailability for in-memory storing. In accordance with an embodiment ofthe invention, each RDMA key-value store write request received in anetwork adapter of a primary computing server to perform an RDMA writeoperation of data to an in-memory key-value store defined in the memoryof the primary server, can be replicated to a network adapter of asecondary computing server. Further, address translation data for thein-memory key-value store of the primary computing server can beprovided to the network adapter of the secondary computing server. Inthis way, during failover from the primary server to the secondaryserver, access to the data of the in-memory key-value store in thememory of the secondary server can continue as if it were the in-memorykey-value store in the memory of the primary server without requiringintervention of the CPU of the primary server.

In further illustration, FIG. 1 pictorially shows a process for RDMAoptimized high availability for in-memory storing. As shown in FIG. 1, aprimary server 110A can implement an in-memory key-value store 120Aaccessible through RDMA operations by a corresponding network adapter130A. Likewise, a secondary server 110B can implement an in-memorykey-value store 120B accessible through RDMA operations by acorresponding network adapter 130B. Optimization logic 140 can executein the memory of the primary and secondary servers 110A, 110B. Theoptimization logic 140 can process RDMA requests from a coupled client150, such as write, read and update.

Initially, the client 150 can issue an RDMA write request 190A to theprimary server 110A to write data to the in-memory key-value store 120A.The optimization logic 140 can direct the network adapter 130A toperform the RDMA write operation for the data and the network adapter130A can return addressing data 160 to the client 150, including anaddress of the data in the in-memory key-value store 120 and an addressof the secondary server 110B. Further, the optimization logic 140 canreplicate the RDMA write request 190A to the network adapter 130B alongwith address translation data 170 indicating the address in thein-memory key-value store 120A at which the data is stored. Theoptimization logic 140 in the secondary server 110B can process thereplicated RDMA write request 190A and can determine a delta between theaddress translation data 170 and the address at which the data of thereplicated RDMA write request 190A has been stored in the in-memorykey-value store 120B.

Thereafter, RDMA update requests replicated by the optimization logic140 from the network adapter 130A of the primary server 110A to thenetwork adapter 130B of the secondary server 110B can be processed bytranslating the address of the RDMA update request utilizing the deltabefore directing an RDMA update for the data in the in-memory key-valuestore 120B. Of note, the client 150 can store a local hash map that foreach key stored therein in respect to data stored in one of the servers110A, 110B contains a remote pointer (i.e. address) to the datacorresponding to the key in the primary server 110A. As such, the clientupon failover to the secondary server 120 may access the in-memorykey-value store 120B of the secondary server 110B directly using theremote pointers/addresses of the local hash map.

To that end, the client 150 repeatedly can issue to the primary server110A RDMA read requests for data disposed at respective addresses of theprimary server 110A. However, to the extent that the primary server 110Abecomes non-responsive, the client 150 utilizing the address of thesecondary server 110B can issue a failover RDMA read request 190B to thesecondary server with the address of the requested data. Theoptimization logic 140 in turn can translate the address of the data toa corresponding address in the in-memory key-value store 120B utilizingthe delta between addresses in the in-memory key-value store 120A andthe in-memory key-value store 120B. Using the translated address theoptimization logic 140 can direct the network adapter 130B to perform anRDMA read operation at the translated address and the network adapter130B can return the retrieved data 180 at the translated address to theclient 150. Optionally, a local translation table maintained in theclient 150 can be translated in response to a failover with the delta soas to issue subsequent RDMA read requests and update requests to thesecondary server 110 with an already translated address.

The process described in connection with FIG. 1 can be implemented in anapplication server data processing system. In yet further illustration,FIG. 2 schematically shows an application server data processing systemconfigured for RDMA optimized high availability for in-memory storing.The system includes primary and secondary application servers 210A, 210Bcoupled to one another over a computer communications network 250. Theprimary and secondary application servers 210A, 210B each include memoryand at least one processor and also include respective network adapters220A, 220B. The primary and secondary application servers 210A, 210Balso include respective storing engines 240A, 240B managingcorrespondingly different in-memory key-value stores 230BA, 230B intowhich application server objects are key-value stored.

Importantly, an optimization module 300 can be included as part of thenetwork adapter firmware of each of the application servers 210A, 210B.The optimization module 300 can include program code that when executedin the memory of the application servers 210A, 210B is enabled toreceive an RDMA write request for data from a client 260. The programcode further can be enabled to direct the network adapter 220A toperform an RDMA write operation in the in-memory key-value store 230Aresulting in an address at which the data is stored in the in-memorykey-value store 230A. The program code yet further can replicate theRDMA write request for the data to the network adapter 220B with theaddress such that the network adapter 220B can perform an RDMA writeoperation of the data into the in-memory key-value store 230 resultingin an address at which the data is stored in the in-memory key-valuestore.

Utilizing the address received from the optimization logic 300 of thenetwork adapter 220A, the optimization logic 300 of the network adapter220B can determine a delta between the addresses acting as addresstranslation data. Finally, the program code can return not only theaddress of the data in the in-memory key-value store 230A to the client,but also the program code can return to the client 260 the address ofthe secondary application server 210B in the event of a failure of theprimary application server 210A. In this regard, in response to afailure of the primary application server 210A, the client 260 can issuean RDMA read request to the secondary application server 210B with theaddress of the sought after data in the in-memory key-value store 230A.Using the delta, the network adapter 220B can translate the address to avalid address in the in-memory key-value store 230B and the networkadapter 220B can retrieve the sought after data in the in-memorykey-value store 230B.

In even yet further illustration of the operation of the optimizationmodule 300, FIGS. 3A through 3D, taken together, are a flow chartillustrating a process for RDMA optimized high availability forin-memory storing. Beginning in block 305, an RDMA write request fordata can be received from a client in a network adapter of a primarycomputing server and in block 310, the RDMA write request can bereplicated to a network adapter of a secondary computing server. Inblock 315, an RDMA write operation can be performed by the networkadapter of the primary computing server on the data resulting in anaddress of storage of the data in the in-memory key-value store of theprimary computing server. In block 320, the address can be retrieved andin block 325 the address can be provided to the network adapter of thesecondary computing server with which the network adapter of thesecondary computing server can compute a delta between the addresses inthe in-memory key-value store of the primary computing server and thoseof the secondary computing server. Finally, the address of the data andan address of the secondary computing server can be returned to therequesting client.

Turning now to FIG. 3B, an RDMA update request an be received in thenetwork adapter of the primary computing server in block 335 an RDMAupdate request specifying an update of data at a specified address canbe received in the network adapter of the primary computing server. Inblock 340 the update request can be replicated to the network adapter ofthe secondary computing server in response to which the network adapterof the secondary computing server can update data at the addressprovided adjusted to account for the delta. Likewise, in block 345 thenetwork adapter of the primary computing server can perform an RDMAupdate of the data at the specified address.

In this regard, as shown in FIG. 3C, at block 350 the replicated RDMAupdate request can be received in the network adapter of the secondarycomputing server along with an address in the in-memory key-value storeof the primary computing server at which the data to be updated can belocated. In block 355 the translation delta previously computed by thenetwork adapter of the secondary computing server can be retrieved andin block 360 the received address can be translated to an address of thein-memory key-value store of the secondary computing server. Finally, inblock 365 an RDMA update operation can be performed by the networkadapter of the secondary computing server on data at the translatedaddress.

Referring now to FIG. 3D, in block 370 a failover RDMA read request canbe received from the client at the network adapter of the secondarycomputing server. IN block 375, the translation delta previouslycomputed can be retrieved and applied to the address of the RDMA readrequest at block 380. Subsequently, in block 385 the network adapter canperform an RDMA read operation at the translated address. Finally, dataretrieved from the in-memory key-value store of the secondary computingserver resulting from the RDMA read operation can be returned to theclient in block 390.

The present invention may be embodied within a system, a method, acomputer program product or any combination thereof. The computerprogram product may include a computer readable storage medium or mediahaving computer readable program instructions thereon for causing aprocessor to carry out aspects of the present invention. The computerreadable storage medium can be a tangible device that can retain andstore instructions for use by an instruction execution device. Thecomputer readable storage medium may be, for example, but is not limitedto, an electronic storage device, a magnetic storage device, an opticalstorage device, an electromagnetic storage device, a semiconductorstorage device, or any suitable combination of the foregoing.

A non-exhaustive list of more specific examples of the computer readablestorage medium includes the following: a portable computer diskette, ahard disk, a random access memory (RAM), a read-only memory (ROM), anerasable programmable read-only memory (EPROM or Flash memory), a staticrandom access memory (SRAM), a portable compact disc read-only memory(CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk,a mechanically encoded device such as punch-cards or raised structuresin a groove having instructions recorded thereon, and any suitablecombination of the foregoing. A computer readable storage medium, asused herein, is not to be construed as being transitory signals per se,such as radio waves or other freely propagating electromagnetic waves,electromagnetic waves propagating through a waveguide or othertransmission media (e.g., light pulses passing through a fiber-opticcable), or electrical signals transmitted through a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Finally, the terminology used herein is for the purpose of describingparticular embodiments only and is not intended to be limiting of theinvention. As used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, integers, steps, operations,elements, and/or components, but do not preclude the presence oraddition of one or more other features, integers, steps, operations,elements, components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

Having thus described the invention of the present application in detailand by reference to embodiments thereof, it will be apparent thatmodifications and variations are possible without departing from thescope of the invention defined in the appended claims as follows:

The invention claimed is:
 1. A method for remote direct memory access(RDMA) optimized high availability for in-memory storing of data, themethod comprising: receiving RDMA key-value store write requests in anetwork adapter of a primary computing server directed to writing datato an in-memory key-value store, the key value store storing associativearrays of data of the primary computing server, each associative arrayincluding a key associated with a single value; performing RDMA writeoperations of the data by the network adapter of the primary computingserver responsive to the RDMA key-value store write requests;replicating the RDMA key-value store write requests to a network adapterof a secondary computing server, by the network adapter of the primarycomputing server; and, providing from the network adapter of the primarycomputing server to the network adapter of the secondary computingserver, address translation data, the address translation dataindicating the address in the in-memory key-value store of the primarycomputing server at which the data is stored.
 2. The method of claim 1,further comprising: receiving from a requesting client an RDMA key-valuestore read request in the network adapter of the secondary computingserver in respect to data stored in the in-memory key-value store of theprimary computing server; translating an address for the data stored inthe in-memory key-value store of the primary computing server to anaddress in an in-memory key-value store of the secondary computingserver utilizing the address translation data; performing an RDMAkey-value store read operation at the translated address; and, returningby the network adapter of the secondary computing server the dataproduced by the RDMA key-value store read operation to the requestingclient.
 3. The method of claim 2, further comprising: translating alocal address table of the client based upon the address translationdata; and, forwarding all subsequent RDMA key-value store requests fordata in the in-memory key-value store of the primary computing server tothe network adapter of the secondary computing server utilizing thetranslated local address table.